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. effects on student achievement. 


— peaiee: may choose, trom these m i” or 2 velop ba acceptable alteruative to : 
J. 
gmmerate Title I stlacte data om at the Federal level. The three. models are: 


4 


th 1974 Congress aniended the law moverales ema Title I project evaluation to 


4 


recipe dompe rable data across projects. These data are to include objective, ° 


meusures of student arom As a result of the 1974 amendment, the United _ 


States Otte of Education (USOE) qoniates cerelopment of the USOE Title ! Evaluation 


and Ropowins System. The System, doveloped by the RMC Research Cokporation, 


applies to Title I projects i avis skill 4 of —aet reading and language 


arts in Pere 2-12, ‘Yaplementation of the, ioe System would standardise Title I 


_ evaluation procedures a ceeds conn: evidence concerning Title I project 


At the beart of the. System are , alternative evaluation models. A local 


In describing cae of fhe models RMC —_— that, if shige imple- 


~ mented; they. will proce: «unbiased ee a a Title: I project t zee atment effect 


... (Tallmadge and Wood 1976). Each aaa provides a rationale for qusptitying the 


expected mean achievement score of the Title I project students at the termination 
of a project periéd under the assumption that the project has tied no effect at all. 
This score, ™ the n no-treatment expectation, Is subtracted from the observed 


e 


mean peep eat ment achievement score of Title | ideas, resulting in the Title I 


project treuument etfect. 


tet ks pad 
ee a Be te er Y 


i) 


" The most a ad model among practitioners is the Norm Flfeenced Model. : 


its popularity is ‘ fouahly due to ease of imp}ementation a the tact that it has little |: 


significant effect on the student selection process. The Norm-Referenced Model 


requires neither random qssignment of students into the Title torectait nor ee use 


of a atrict cutoff score on.a ndiections meanire. . 


e 
; a " r . . 
‘) . . 


" Selecting students le an inportint contact point between the operation of Title I 


projects and the implementation of Title I evaluation, Students who are in greateat 


4 ’ 
; 


need of services are the highest priority studeats from the point of view of program , 


operation, On thé other hand, the’ process of selocting students into a remmnent: 
grow must be clearly defined in’ er to avoid blas lia aca tae effect miesgures. 
standardization of sieceiasas for student selevtion must be viewed froth these two 


points of view. 


® 


The procedures for generating the Norm-Reterenosd Medel no-treatment N at 


3 
a 


axpsoiation are: a ‘ 
. ds Select students for the Title 1 project:on some measure other than ‘cais 
protest score, , ; oe 
2. slid iad Title I students win a a test at a time corres= 
| re to empirical norm mi dateas . | 
3. Compute the mean pretest standard score for Title 7 students aie de 


Students tas do not have posttust scores). 


¢ 
o 


‘ 4. Convert the mean pretest standard score to Its satouiila equivalent. . 


. 5. Define thy percentile equivalent of the mean pretest: standard score a6 


the no-treatment expectation for i Title I group. 


4  .° 


&, 
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Separating the pretest fron: the selection a ls intended to contro! regression 
effects. , If this contzol 1g inadequate, then the equipercentile assumption applied to - . 
" . . vee appears to be false. i 7 3 7 : ‘ ; Bo ee 
| ; Initial work addressed the’ question, 'Do ‘Independent administrations of the 
a : selection test and the pretest control for regression offects in the Norin-Reforenced 
. Model ‘when the mean orem of the selected group is used to aa the no-treatment, - 
so expectation ?' ‘Analysis implied that ‘Implementing in Norm-Referenced Model may es 
: result in a biased treatment effect. Independent selection and pretest measures | 
* reduce ee not necessarily Control for regrossion. Bias Is Indicated when at 
« -oenalitiog between the selection measure and the pretest is different from the . 
_¢orrelatton between the eelceica measure and the posttest Prox Ytxicy Where: 
ee denotes selection teat scores obtained at occasion 0, Om 
X) denotes pretest scores es ongined at occasion 1, and . 8 
‘Xo. denotes posttest scores‘obtained at occasion 2. | oe 
= most paneey the inequality will be. Txgny Magne: The, selection te test wil correlate ° 
baie with me pretest than It will with the —, When Tox1>Txoxg? regression | 
- ; . blas will rem even though the selection bea scores are obtained independently o the. 
| pretest scores, “The direction of.blas will jaa in an overestimate of tiie Title 7 
treatment effect. - ; . 
After demonstrating het the Noia-Rete renced Model would likely produce biased 
| treatment aca measures, four possible actions were cons{dered--they were: 


e 


1, Recommend drépping the Norm-Referenced Model from the = 


, 2, Recominend treating results of Norm-Referenced Model e uations 
-_ . 
\ ok 
@ ; nana " 


ai 


separately. trom those produced by the Control Group Model and the | ~_ 
Special Regression Model and d cual conclusions reached in Norm- 
Referenced evaluations. , 
a | a Encourage SEA's and LEA's to choose another model, 

Cat Pevelop a sounder method for oman the Norm-Referenced Model 


no-treatment expectation, 


’ 
¢ . 


The purpose of present. analysis (s to provide the’ theoretical basis for improving — 
“the no-treatment expectation satiation procedu-ei tor the Norm-Referenced Model. . ; 
. The ‘method used to derive the alternative definition of the no-treatment cepacia is 

% based on a structural equation model. The model developed is an n extension ‘ofa 
| model nace ats) developed for analyzing nonequivalent control group. designs, 
‘Stmuctural equation nodela~~ip this case a recursive path analysis model--can = 
| used to devine the appropriate no-treatment expectation for the nonequivalent ee 

group dasign where student selectiolf has been based on a separate measure of the 

'., game achlevement construct measured by the pretest and ere measures, The 
poodel.. which may also be. valled a causal model, includes a clear statement the 
distributional assumptions and causal ordering ot _ viriablea sssumed to usidetile ay 
the measures obtained in the — The opusal model used to matte an altern- ‘ a 
tive no-t reatment expectation for : Norm-Referenced Model does not require the 
assumption of equlpercertile ata or the assumption that regression ls controlled 
by the dual pretest dealgn, 

In order to intulively grasp the nat rematical argument "ee flows, consider 


o @ rae 


‘ Kenny' 8 , ddvice that: 


"the researcher carefully atpdy the process of selection into treatments. , 


é 


The critical questions are: (a) With what and by how much does the treatment 


correlate with the causes of the dependent ule (b) wien does 


selection take sasee The answer to enese ices fiesarriiaga the mode af 
analyais: " (page 359) | ‘ - 
In other words, if the process of selecting Title I students from a known population | 
(or pbplatioas| can be quantitatively modeled, one can ratlonally generate a ; 


no-treatment expectation. 


Al 


Random assignment of students — a single population ts a srlentine process 
= well mon properties. r supplies the theoretical basis for the Control Group 
Model no-treatment expsctation with no adjustment andthe analysis of covariance 
es Random (or complete) assignment of students to Title I from one popula- 
‘tlon and random (or complete) assigument of students td a Control Group from a 
second Goat provides the theoretical basis for the pencipal axis adjustment in 
the Control Group Model (Kenny, 1975). 

Likewise, selecting Title I'students from a siagie population. on the bagi ofthe ss; 

' best available covariate (0.8. the pretest score or a componite measure ibitoding ? 
? ra) =e 
the pretest score) préddes. theoretical justification for establishing the Speci 
. . . Hegression Modal no-treatment expectation, "Theoretically, the adjustment applied | ) 3 
in the Spectal Regression Model is identical > to the analysis of coyariance adjustment : 
for the Control Group Model. However, the procedures for estimating the adjustment ) 


ce, 


factors are cuearaegiahed 
Maples siweeating the scestahacts derived by Kenny, treatment effect ooeeela 
,tlons need to be briefly explained because the structural equation model includes the 


treatment vartable aga ‘dummy variable, . 


of 


Treatment E ffeot Correlation 


ha To understand the statistical arguzhent tn this paper It will bb necessary to 


" conceptualize treatment effect measures in correlational terms aa well as mean » 


bh, : ; oo 
difference terms. The null hypothesis for an evaluation mode! that contrasts a 
+ . 


treated group with an untreated group can be expressed as the expected polnt-biserial 


“~—— 


correlation between the treatment (e.g., coded as a dummy variable where treatment 


group memberg are scored as 1 and control group members are coded as 0) and the - 

posttest score. | magnitude of “ correlation is . fumetton of (a) the mean i ~_ 
difference between he treated group and the untreated group on the continuouagly 
distributed pretest rc posttest — (b) the piuooeia of subjects In the treated or 
untreated group, (c) the standard deviation of the. combjned groups on the 


continuous vartable, 
. Two alternative formulas for the point-biserial correlation are: 


He 


2 fe 
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eee . 
> 

xe ren , : 
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r #%,-X ) | - 
ptbis  “t ; - 
ae —— nt | (2) 


y Where: 
t represents the treatment group, 
c represents the control grou, - 
§ represents the combined treatment and control group, 
p represents the proportion of subjects in the g vho are in treatment group, 
q represents the proportion of subjects in the g who are in the control group, 
_ Og’ Fepresonts the standard deviation of the g On the continuous vartable 
(Le., pretest or posttest), . : 
%, represents the mean of the treatment group (t) on the cont tnuous variable, 
X.. represents the mean of the control group (¢) on the continuous variable, 
X, represents the mean of the combined group ig) on the continuous variable. 


eee O.. 8 | , 
ys ERIC \ ’ 


| | aes : 
The ne of a treatment can be assessed by comparing trvo treatment effect 
corelatiGauscine {lrst Is based on treatment status cotrelated with pretest status 
(l.@e, the p.etreatment otfect correlation) and the second is based on treatment 
status correlated with posttest status (l.e., the posttreatment effect correlation). 
; ; Assuming that 3 aad q remain constant, a treitznon effect may be ee by 
a more ssaliive (or less negative) posttreatment effect correlation compared to the 
pretreatment effect correlation. | ) 

When a randomized experiment has been iplauedael euseneatuily and the 
means of the treated group and the untreated group on the pretest are equal, the 
pretreatment effect correlation is zero. In oxpectdncy terms, the expected value 
of the eee effect correlation under ranean assignment is zero. 

| whee vaadean titi has failed to sila equal pretest means due to chance, | 
or where selection Is pased on factors correlated with the anton: the ra 
effect correlation mil be nonzero. If Title I student are selected on the basaret 
low ‘educational achiavonit and higher achievers make up the comparison a 
the pretreatment effect correlation (Tx, 7) will be negative. 

A pctceatmnnd sepeccitiens ora ull. hypothests, can be eioeesied as an 
expected value of the posttreatment effect correlation itygt)s An oxpected null 
posttreatment effect correlation can also be converted directly to an expected mean 
value on a posttest measure for the treatment group, ussuming the treatment had no 

aa: sa. other words, by solving tys Ro in the formule for a point-bisertal corre- 
lation, the no-treatment expectation can be expressed as a mean posttest score in an 


appropriate metric. The regression of X, on T is used to predict the vale of Xo 


for those in the treatment group. 


eS . 
~~ 


as ; _ eo ; 
Solving for Rp Is iustraved in the felling example. 
t u 


Using formula’ a, . | 
{ ° 8 ° . ~ 
yt Su Sel ae, 
: Cg q : (3) 
- Where | represents the occasion ret testing (i.e., pretést or posttest), 
' where mal represents the mear. score of the treatment group on the continuous — 
_(e. q. ’ mean poemet score of Title [ grotp), . . a 


where Rgi represents the mean score of the combined groups on (* :ontinuous variable _s 


 (@.g., mean score of local norm group taken at the time of the posttest). 


Solving for Roy: - 2 
hes | , =< na 
xoT ™ Xe - Xe0 of _ ; 3 

_ +. . _ : —_ 

. od OG 

— xgToy * Reo - Xe. ; 


ee Me | 
2g g 
a a (4) 
Given  pabesad aa of rg, an emectad va of Riz may be computed under | 
«the nual hypothesis." 


‘Example. Assume a hypothesized no-treatment effect correlation with a value of: 
; Ss FxoT a’ =,40, 


” ‘This value means that the treatment group is expetted to score lower on n the posttest 


\ 
than the combined group (e.g., a local norm group). ° 


Second, assume. a value of 21.06 for the standard deviation of the combined group 


on the continuous achlevenrent measure: 


es 


gm 21.06. 4 (y 


Third, assu:ne 30 percent of the students la the combined group are in-fhe 


treatment group and 70 percent of the students are in the nontreatmont group. Thus, 


p= .30 -_ q=.70. 
, Fourth, assume that the mean score of the combined group at the tline ot the 
posttest is 50 NCE's (a standard score used In Title I Evaluation and d Reporting system 
Keo = 50.: 
- 


Then, in solving for Ro. the predicted posttest mexn for Ee tren mee group 


Vv 
ihe treatment has had no effect, we have: . ae ° 


Rp = AO (B.08) , 5 0 882. 50 = 37.1. . 
od /.70 . 655 ‘he 7 


The value 37.1 Is the no-treatment expectation expressed as site predicted mean of | 

, the trpatment group oh the posttest. ; | | 
This example Ulust rates how a hypothesized no-treatment expectation expressed 

asa point-bleerial can be converted to a ae tveibniend expectation papraeeee as a 

mean. Tie Title I no-treatment expectations, then, can be sions (under specific 

conditions) either asa mean or a poatt reatment effect correlation. ‘ Therefore, the 

proof included in this raper applies directly to tha Title I Evaluation-and Report ing 


f 
System convention for expreasing the no-treatment expectation. 


GendFal Formulas for Expressing Null Hypotheses 
As Indicated above, the appropriate no-treatment expectation for the randomized 


experiment is expressed as follows: 
; 
Ho: Tx)T © FRoT 2 0; 
or, in terms of mean scores:. 


Fg! abyg "Abe o 


ff 


li 


iets , 
: : a ‘ 
fe  ® “4 
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The problem arises when Tx, THO, Bare SieRep. wl which may be the result of 
chance or a nonra selection process, A statistical adjuniont may be otmployed 
to generate the appropriate null hypothesis in this instance, ; _ 

Kenny (1975) has expreased the null bypothesis associated with each of four 


Pal 
common alternative estimation methods. They are presented In Table 1, 


TABLE 1 
: Null Hypotheses Corresponding to Four Statistical Estimation Methods 
" ESTIMATION METHOD NULL HYPOTHESIS ~- 
ANCOVA 7 ExT * Px TMi xp 
ANCOVA with Rellability Correction TxoT * Px, TTx)x9/Tx1x) 
L. Raw Change Score Analysis TxoT * Fx) Tx, Wrxo 
“Standardized Change Score Anslysis TxoT "IxiT 


Using a.genersi linear model and a path analytic logic, Kenny specified three. 


. = 
alternative selection processes and derived the appropriate null hypothesis from each 
A 


ofthem. These seiection processes and toeir associated null hypotheses are presented 


, ' . 


In Table 2. 


TABLE 2 


Based on the Protest 
i 
Based on Pretest True Score 


TxoT * "xy T xy x9 


TxoT “ Px T8x) xo/ Txixy | 
Based only oa Group Membership TxoT bl xT 


Between the Pretest and Posttest Tx,T «Tx T a , 


Kenny's model ca be used to Sega expectations for the 
Control Group Model with is sijustmanty the ANCOVA adjustment and the principal . 
axis adjustment. It cn also be iad to generate the no-treatment expectation for 
the Spectal Regression Modét, Since the Kenny model Includes only one pretest: 
acon, bowever, It a ee to derive the no-treatment expectation for the 
Nora-Referenced Model. In this paper,’ a cece pretest varlable and two 
unmeasured variables are added so that © no-tre:.tment jaoeccaticg for the Nortn- 
Referenced Model can be derived. Thus, the basis for the Norm-Referenced Model 
no-treatment expectation is made explicit ts its definitions and urfderlying assump- 
tlons. Moreover, the newly-defined model also provides a basis fee comparing all 
three Title 1 svat ‘on models (including the three adjustment ditemmarivea for the 


Lontrol Group Model). —_ . : e 


t 


f 
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Derivation of a No-Treatment Expectation for the Norm-Referenced Evaluation Model 


While the Norm-Ref« renced even Model ces not oe: stendardéd 


procedures for selecting students tnto the Title ee It is a common practios 
use soine measure of sso student achlevement in ‘the selection process. This 
siastice \s supported by Title { regulations and recommended by RMC. It would be 
informative, then, to develop e formal paychometric and ceusal-mode] that would 
result in a io-AeeAtmant expectation where subjects are selected Into Title [ on the 
basis of a measure taken prior to the prevest.. The model includes four measured 
variables --they are: | | 


\ 
Xp # an achlevemant measure used for selection, 


r 


X, = a pretest meagure of achievement, 
Xo a posttest eam of achlevement, 
T = the treatment variable. : 
The causal function of X is based on three mneneies thet ical constructs: 
G = group thembership (e.g., sex, race and eee 
Z = individual differences within groups (e.g. , ability, as hievement), 


Es errors of measurement or unstable causes of Z. 


Assuming al) vartables have been standardized to reduce algebraic complexity, 


we can express X, and Xp In terms of their causes, G, Z and E: 
Xq. * 2G, + boZoy . gE gy: 
Xqy 7 ayG, + by 21, + ep Egy, 
Xoj * 89G) + bo2o) + e2F a, 


\ 


where the subscripts 0, | and 2 refer to occasions, and uvbacript | to the subject. . 


to . 


(5) 
(8) 
(7) 


Note chat, since we are deriving he no-t reatment expectation, no oe on Xu are 


s 


assumed to result from T, the treatment variable, 
‘ ’ a lf . . : 
| The following assumptions are made for thig series of linear equations; 
Y . y ; . : 
. 1, Group membership (G) 4s assumed to remain sonstanl across time; 
oa 


its autogorrelation equals 1. 00. 
2s Relative position within groups 2) may vary across time; 
its autocorrelation may be less than 1.00. 
| 3. Errors of measurement (E) are perfectly unstable, or random; 
| its autocorrelation Is 0. 
‘. Au unmeasured varlables (G, Z and E), are assumed to be uncorrelated | 
with cach tay, 
5. Gswp # cavallwadl and relative scsition within groups (Z) do not 
interact, -_ my | 


; : : : e aes 


The first two assumptions mean that ap individual's status as a member of some 


group remdins constant over time, while status on individual tratts varies over tise. 


According to the model, Z ts defined as relative position within groups (e.g., standard 


“scores based on differettiated norms would represent a Z), and errors of measurement 


‘ 
4 are defined as uncorreiated with other errors of meateeement, as well as with G and Z. 
It follows then that there Is no confounding of effects of the unmeasured variables on X. . 
Finally, the assumption that G and Z do not Interact Is analogous to the assumption of 


homogeneity of regression in analysis of covariance,’ I means that the regression of 


X on Z Is the same for all groups considered, ; 


: . = aS e ie ae ; 
ERIC.“ . 2° a _ 


If the isateieul is correlated with the selection pial then It must be confounded 


with the causes of the scores on the selection test. In other words, whether an 
NY 


individual is in the treatment group or the control group | is a lon‘ of his pane . 

membership, bis restive status withiathe. ‘group, errors of Vaan or some 
combination of the three variables. Given this, the causal function for the ireunedl 
variable ts written as: . 

Ty, qG, * mZq, + Eo + 10; 7 the (/ 
where U is a residual — that is ince erelated with all unmeagured variables. The. 
attalior the residual term, the more reasonable the model that sale treatment and f 
sont rl group membership to be based on the same variables that determine pretest f 
status, A ants residual term would iepiette that ase(yament | to treatment is : 

upon variables that are ufcorrelated with G, Zand E. Since these variables ary 


$ 


unmeasured, a direct assessment ot the residual term is not made. y 
~ The gext step in developing the no-treatment expectation is to analyze oe 
corre lationg-between the measured variables Xo, X1, Xo and T. To do this, we first | 
- expross the sovrelations (which ate eneerrepie) in terms of the naotarend partial 
neg reealon coefficients (which are isobeatvabls) specified in equations 5, 6, 7 and 8. 
The principles of path analysis ald in this process... Accordingly, have provided 
a path diagram (Figure 1) for the model spec ifled in equations 5 he 8 and the 


assumptions which support those equations. » 7 


/ 


If; 


Sent 
‘ 


e - . e w . 6 e 


Soh migurei foe we 
Path Diagram Depicting the Application of . 
. Independently Administered Tests for _ 4 
_ Selection (Xo) and Pretesting (X;) 


i? F . es ‘ 
’ 
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Since G, 2 _ E are latent variables, the Wersatin coefficients taking tows 
variables vith the measured variables Xo. X1, Xo and T are also inobasrvable, 
However, ~ making certain asrumptions “a taking advantage of meenrale Westin 
we can use these coefficients to show theoretioally-expected relationships among the 
—— variables. 


e 


The correlations between the observed variables + (Xo, Xi, Xa am T) are i 8 Weseg, 


expressed ‘in terms of the regression coefficlents ipath coefficients) ln Figure 


, These correlations are: 


Fxg * aig + Miby + sag we 
. ‘Ex ,T © 481 + mbb; | . e. ) (10) 
| / age = ug + mjby any 
Ro " a Frege * Aon + bobyh _ 7 7 aay 
2 Rgeg" apag = bobs) “45 4 aa), 
. : Faia Mate + Dabal Ho (aay. 


In ‘each of these equettons, the correlation between two variables | is taken ae the eum 
of the. produats of certain "path coefficients. ” For those who are more —— with . 
multiple regression techniques, each of these path coefficients can be shown wae 
: conventional bie as standardized partial regressiin coefficients. 

Having jaieeeed the correlations between the measured variables tn terms: of - 
path coefficients, we now proceed to further specify the model in terms. which will 
ay allow us to express expected relationships between the observed correla- 


. tlons. What we are moving toward is a way to express rgoT as a function of Ty? 


18 


® @ ms 
one “of selection | is based upon ° we have: ‘ : 
| . 2.8 & “8515 + WX gy i Uy. on “ as) 4 = 


| atic expresses membership In the treatment group asa function of the selection teat 
score (Xp) and a. rostdual (U). 
Next, we can substitute oweion Sefor Xo; In equation 15, which gives: 
Ty = k (ag@, + boZo1 + egEo{) + 10. 
Since T also equals Wy + mZq + 8Eo; + HU; from equation 8, we have: 
ak a bok = m, ‘egk= 8, | 
. --OR-- 
\ rs] 2 
ag" M/by * 8/eq ~ k: a _ 3 
‘Knowing that a2, + b2p + 0%) =1 because - is standardized, It is possible to show 
Tx T |s equal to k: =, a | a | - 
at "to * y+ at (funtion 9) . 
Fro" . ty + Poly ” “soley = 


” Next, we show thi Ty, T* kry x, since q = agk and m = bok: 
"xT  agka; + bokhb, (equation 10) 2 
ty TP? Kelaghy + bodyhy eG te 

Part * KPxoxy a a = 


Now thet we have CxoT @ k and Tx, T . Kr xox! we can substitute for k in Kr xox) 
and find: | 


FeyT * Fok * FRgny: 


: the selection oat and the protest, Since Fon, will neariy always be leas than’ 1, 00, ~~ i: 


e 


This expresses the correlation meeneen the penal and the treatment In terms of | \; 


the correlation neeweee. the selection tent and treatment and the correlation between - 

.W@ Can-expect tT to be lesa than _ The equation eiT = reo * aon is 

ldentical to aa me hypothesis for ANCOVA if x were the posttest rather than the , 
"pretest. oe solution to this point is: ; Went ioal to the solution offered by Kenny. (1978) 5 
page 350. Since the logic of Model A requires that we solve sd TxoT in terms of . 

Tx T° additional steps are needed to find these results. -  t 
_ We begin by showing that: }: 
“> ° | 4 "ey = agkay + bokjbo, . 4 . 
which follows directly from ry,7 * 82 “+ mijbo, and q= ak and'm = = bok : 

Factoring k out of agkag +, bokjbg gives: . & : - | 
| *xgi “ =k (agag + bald. | 
“Knowing that: i ; ) / my 
| Txoxe * 2982 + bolbo” (equation 13) | 
Fyn boy 
Substituting for k gives: : 


leads to: 


“xgT,” TgT ie . . 
but, since MoT * Tx, T/Txox): because TxiT * TxoT ' Txoxy: We have: 


TxoT * eit (Trax 


2 : 


Ga a Cc St I SR a A sal 


This final expression gives the expected value of Tyo where: 


” 


(1) A selection co has been used to assign atudants to treatment; 


(2) An independently oe test hs bpodlused & as a pretest, “ 


a 


which is then used to cenatiie, a ibesaltne’ or no-tvesiment expettation. - 
. The ratlo FOXD/ tees, is the —— statistical adjustment for rx 7 
aseum ing that regression le biscebe uous and linear. 
. j 


An Example Comparing Alternative No-Treatment Estimation Methods for the 
Norm-Referenced Model 


A Title I program ae schedule may —— « selection test in the spring, 
a pretest in the fall,'and a posttest In the following spring. Table 3 shows the uses. 


that te alk be made of these test data in a Title I evaluation. i . 


- GABLES 


> ite 
Use of Test Data __—- . *, 


. Spring '78 |. selection "‘and 
: . pretest 4 
: . selection - : 
Ce = TE 
vee : 
Se 


Each column represents a plan for test use. The pretest Is used to generate 
: 4 .. 


the no-treatment expectation in each of these cases. A separate selection test is 


applied in Plan 1. ° 


2h 
21 | 


i 5 2 neat ee ; mn i 4 . > ‘ 
PE | es ee a ee. ef eee ae eT aes 6 a TN ae | ee er es a ate a! ae ee er en ae ee ek or 


; 
=~ ~ 


Three statistical estimation factors that may be applied to these plans are: 


Cg , 
. : . 7 Py ~ . 
. igsoume variances are equal so Cx, lore = 1.00); - 
Py ‘ . , . 
08/1 | : (Method B) 
RX" , ; ata C) 


Method A sesaptads to the standardized gain or peiholpals axis en factor. 
When NCE (Normal Curve Equivaient) scores are used (ande,, e x,)s Method A 
carrenpeons to the equipercentile “— estimation technique which RMC recomnisdds 
for the Norm -Referenced Modal, ; 

Method B is the statintieal estimation factor Serived in this ‘paper as the method of 
choice when selection has been based on an achievement as other than that used asthe . 
pretest. Method C, on the, ctber hand, ts the estimation aioe defined by ANCOVA. 
The ANCOVA estimation factor Is applied bere, as It-has been recommended as an 
- alternative to using the equipe reentile <padeietiog In the Norm-Referenced Model.: 

| Since Plan 1 ls the Norm-Reférenoed design recommended by RMC, the most , 
significant comparison Is between Method A and Method B applied to hen hos 

Suppose we had the following correlations as population paraiuelave: 


Ly 


*o%) 


r = ,70 / 
2  ' 
a = . 80; ‘ * a 
Sine: XyXo 


Ze 


< Mee Pe tah ithe Sa dee tat tee egestas ta, a chgn oe 
. Fee ae a eee 


_ where Xo Is Spring '78 test score, ” ‘ 
xX) is Fall '78 teat score, 
- - ; X, Is Spring '79 test score. . 
" Next, suppose we had selected aideeks on X so that the mean score of the . 

eelected group was one standard deviation below the population mean, The mean 
of the selected group in terms of Z scores and NCE's would be: ; 3 
2 = -1.Q0 (an NCE of 28.94). : 
“If we use 25 and r in assum = to previde the estimate ’ 
» we find: ; a | 7 : 
' @-.80 (an NCE of 33. 18) | on, i 
a re: 
. Given the values of Trg! Tem Fa nat 7 ;an 2, we can apply Methods i | ! 
. _BandC to Plan 1. The results are presented in an 4. eo = 
“TABLE4 . | . 

Method Cc 

‘ 


The no-treatment expectations in Table 4 were found by applying the following 
two formulas: — | | 


Bins as Estimation Factor : Bx! 


ubere Bi is the expected mean Z score of the treatment group on the pretest, 


as Bo. | ; a, 
E we a ' 23 ; ‘ , ; : ‘i 


. 


Then the value of Bye mas enemrnad to an ANCE by: 
Mean NCE of i 
Treated Group, - * 50.00 + 21. hay ). 
" assuming no effect 
_ The ‘results show that the equipercentile growth method recommended by RMC 
underestimates the no-treatment expectation by 2.11 NCE's. On the other hand, 


ANCOVA with the pretest as the covariate overestimates the no-tryatment expectation 


by 1.26 reuse prior selection sn Xp was ignored in choosiug the value of Tx 1% 
the satieaseion factor. 


The amount of regression from the selection test to the pretest Is from 28, a to 
33.15, or 4. 21 NCE’ s.» The amount of regression from the pretest on posttest is 
from 33. 15 to 35. 26, or 2, a NCE's. ame RMC method sccounted for only twosthirds 
of the regtession, whereas alternative Method B accounted fer all the regression. 

One could argue that Plans 2 and 3 also me considered as possible In the Norm- —_ 
Reférenced Model. Table 5 iowa the regult ot applying: ‘Method A (this method Is not.. 
neccmnmenaed ili RMC unless a _ selection test has been — and Method C. 


spe hae oy dee te See As ete Sak Mase, | aims 


Since selection was based on the pretest in Plans 2 and 3, the mean pretest 
score of the Title I treatment group In Z score terms is -1.00. However, the 
occasion for the preteat In Plan < is the Fall of 1978, whereas the Secaalon for 
the pretest tn Plan 3 Is the Spring of 1978, | 

: Method A is not recommended for use in Plans 2 or 3. in this example, 
Method A compared to Method C tllustrates the amount of regression operating 
across two time Intervals. Method C applied to Plan 3 gives the same no-treatment 


expectation as Method B does in Plan 1, as expected. 


Summary and Discussion | 


Using a meagure other than the pretest score for selecting students, as !n the 


Norm-Referenced Model, reduces: but does not control regression effects. if = 


’ g 


selection measure ‘s mere bad correlated with the pretest measure than it is 

| “with the posttest measure, , additional regression occlre ‘between the preven and 

me the posttest. When an extreme group has been scgced for treatment, the effects . 
of regression will blas treatment effect mensures ‘unless appropriate methods for , 

: estimating the nasiveibneob ex expectation are saya | 

| As It is applied in the Norm-Referenced Modal, the equipercentile growth 

assumption does not provide a sound unblased estimation miethod.. This follows 

directly from proof that the dual pretest procedure, with one test used for selection 

and the second used for estimation under the equipercentile assumption, does not 


‘control for regression. 


1 3 ye 
/ | 
Having establinbed a problem with the dual pretest design, four general 
solutions are suggested. Possible solutions to the problem include: 
| 1, Recommend dropping tha Norm-Referenced Model from the USOE Title I 
Evaluation and Reporting System. _ | 
2.- Recommend that results of the Norm-Referenced Model act be 
aggregated with poe of other models and that conclusions based on the 
Norm- Reterented Model be subject to special qualifications. 
3. Encourage SEA's and LEA's to choose 2... .oer model. 
4, Develop a sound method for estimating the torustelvanced Model 


" no-treatmont expectation, 


The meneral solution presented, in this faper is to develop a sound method for 
estimating the Norm-Referenced Model no-treatment expectation. Recent, work: on 
ciasineperimentdins stresses that fallure to randomly assign subjects to treatment 
and control groups does not necessarily leave dne unable t estimate treatment 


effects free from bias, Where the selection process is known to the extent that it 


‘ 
‘can be mathematically modeled, it is still possible, in theory, to provide unbiased ; 

““ estimates of treatment effects (Cain, 1977; Rubin, 1977; Kenny, 1978; Bryk and 
_Wolsbere, 1977. Cronbach, ‘Rogoga, Ploden and Price (1976) point out that It Is 
lite difficult to effectively control the parameters which do affect bias, 

The specific solution derived Ie theofetical. 2 artis where student selection’ 
is based on a riadins of the same construct as a separately administered pretest 
and posttest. While the analytical’ model used to derive the no-treatment expectation 

J. 


inrolyed a two-factor model sidriying the achlevement measures, ai same solution | 


follows from a sngle-actor model where al group membership factor is Ignored. 


6 2b ; = os 


The tmportant empiricul facts that the model takes into account Include the 
= » . . 


"ext remity of the treatment group mean from the population mean ird the extent of 
temporal erosion. | 
: The first emptrical consideration, the extremity of the treatment grep, can be 
measuréd by the pretreatment effect correlation, tx, T If the pretreatment effect 
‘Correlation 3 an, the treatment group Is not extreme ard regression effects would | 
‘not be-expected. in other words, lf the treatment group's pretest mean Is the same 
As the population's pretest mean, then no regressicn effects would be anticipated, 

The second empirical consideration, the amount of temp»ral erosion, Is - 
indicated by *ha ratlo Txox2 Tes The greater thé amount of temporal iene. . 
the lower: the vahinst thls ratio and the greater the expected regression. When there. 
is no temporal erosion, the value of "xox ‘roxy will be 1.00 and no regression , 
would be expected, regardless of the extremity of the treatment group. The effect of 
the ratio is to apply the apprvpriate welght io "xT In estimating the expected value 

These observations enpty oaly to the dual pretest design implemented as in the 
RMC version of the Norm-Referenc-d Model. | 


Measuring Treatment. The analyticil model used to derive the alternative 
no-t seuluecs expectation presented In thie paper asdumes clearly defined treatment — "os 
and comparison groups. Ifa standard nonequivalent ecmpacinés group design ts 
used, then iw groups can bu Identified from a single population anc the measure of 


T is straightforward. The Norm-Poferenced Model, Beavers 4.6e8 norms tables 


s} es 


ag a source of soodlatisnwaraniste estimates instead of a concrete comparison 
oii. This is probably seen as a practical virtue of the Norm -ReferencecsModel. 7 
- The practical ‘advantage takes a toll in the assumptions eonutzed in order to make . 
the Norm-Referenced Model workable. | | | 
The measure of the pret reatnient effect corner ica from which the estimated 
no-t reatment effect cee Is had simply contrasts the treatment group's 
pretest mean to the mean of the population from whtoh it was drawn, ies lies the 
rub. The choice of norms determines the ext remity of the treatment group. If the 
Somat group can be considered a sample from the population represented in 
atioas? norms (or local sSieiAi, then thee natlonal (or local) norms supply the 
Ssesitat ion parameter estimates needed. These parameters include mean scores 
and standard deviations for the two points In time corresponding to ha pretest anc 
ccationt dates and the s! -pes of the regression lines predicting — scores ime 
selection test scores and posttest scores from pretest scores. 
Thus, the problein of choosing ide norms (e.g., local versus national 
norms} is a problem of determining appropriate procedures for estimating the 
_ parameters needed, It Isa theoretical or logical problem as well as an empirical 
problem. Using an achievement measure to aerery treatment students phetaeny 
guarantees that an extreme group (one whose mean differs from the spalaiton mea 
will result no matter bow complex the definition of the population Is. 
" Use of Ex Post Facto ANCOVA. The dual pretest design precludes applying the 
equipe rcent (le growth assumption as a quantitatively simple procedure. for estimation. 


Consequently, we must question the need for the separate selection test. Might not 


° . 


the pretest be used for selection and estimation of the no -t reatment expectation, 
~" using Tax to make the prediction of TxoT? The answer, reas students are 
drawn from a single known population, .ls yes, but this answer holds only.if prior * 
‘selection (l.e@., prescreening) has not taken blade: The analysis in this paper shows 
- “ANCOVA to underestimate treatment effects wben prespreening ts ignored. Multt- ° 
enw selection processes must is taken into account in estimating. : | 
| Using norms could represent an “ee practical epproach to estimating the 
treatment effects of educational programs. The procedures for sound use of norms | 
must be carefully thought out and studied. “Development of the moe in this paper 
» (sa further st -p in improving theoretical understending of the Title I Evaluat{on 
and Beporine Syatem. The assumptions behind the model ate, parsimonious. 
hey may not be met in practice. If they are not met, then we would expect 
additional erto.: to enter into the —— Yowever, departures from normallty, ; 
° linearity and eanceenelty of regression will not likely make tt procedurally or’ 
rele easter to estimate the no-treatment expectation tn norm-referenced 
crasl-enperimiontal design. In all probability, such departures will make, 


eatimation procedure more complex. 


af 


References. 


® 


Rryk, A.S,, and Welsberg, HI. ‘Use of the nonequivalent control group design when 
subjects are ene Paychological Bulletin. 1977, 84, pp. 960-962, 


Cain, G.c, Regression and selection models to improve nonexperimental comparleons, 


In M. Guttentag (Ed.}, Evaluation Studies Review Agha). Bevesty Hills, 1977: 
Sage pubtioe: lone: 


~ 


‘ Cronbach, L.J.;Rogosa, D.R.; Piodon R. E.; and Price, G.G. Analysis of covariance 


In nonrandomized experiments: Parameters affecting bias. Stanford Evaluation 


“ Consortium, ‘Occasional Paper, 1976. ° ~~ 


Kenny, D.A. A quasi-experimental approach to assessing treatment effects in the 


aonequivalent control group dewign. Psychological Bulletin, 1978, 82, pp. 245-962, 


Rubin, D.B. Assignment to treatment group on the basis of a covar tate. , Jourusl of 
Educational Statistica, 1977, 2, pp. 1-26. | 


Tallmadge, 'G.K., and Wood, C.T, + Maar ldo, ESEA Tie Leteltion sd rmortas 
" system (revised). Prepared for USDHEW/USOE Office of Planning, Budgeting and 


ry 
‘Evaluation. Mountain View, CA.: RMC Research Corporation, December 1976. 


yy 


31 


' : 
. : . 1 m7 bd 4g ptt: * ro ut y 
Bo ae See eho in wate ab ea ae PR We AI es es oe ap et ase ee ek a a ON ag Se eg ae ES eh iene, Sea Me dep gs OB Adee aca Nm, Bad 


